{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W-y43V2AWKlt"
      },
      "source": [
        "# 🦛 Chonkie: RecursiveChunker for PDF and Markdown Chunking\n",
        "\n",
        "![](https://drive.google.com/uc?export=view&id=1FHBnC4CgPjrvSvoXwJiZqg56I8Ck3SbD)\n",
        "\n",
        "In this notebook, we go over how you can use Chonkie to quickly parse, chunk and run RAG on PDF and Markdown files!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fJ5tsfOeRa7z"
      },
      "source": [
        "## Overview\n",
        "\n",
        "Our entire workflow can be summarized in four simple steps (See table of contents sidebar to jump to a step):\n",
        "\n",
        "1. Convert PDF document to Markdown\n",
        "2. With Chonkie's RecursiveChunker, chunk our markdown document\n",
        "3. Prepare Chunks for RAG\n",
        "4. Use our generated data with an LLM!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pghssFC5Rega"
      },
      "source": [
        "## Acknowledgments\n",
        "\n",
        "We use some awesome services in this notebook, be sure to check them out too:\n",
        "\n",
        "- Chonkie (of course)\n",
        "- Arxiv (for our example PDF file)\n",
        "- Docling (for PDF to Markdown conversion)\n",
        "- Model2Vec (for embeddings)\n",
        "- Vicinity (for indexing)\n",
        "- Together Client (for LLM calls)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mv5WRnWmWKlu"
      },
      "source": [
        "## Installs and Imports\n",
        "\n",
        "Installation might take a while, but only the first time!\n",
        "\n",
        "**You may have to restart on your colab instance.** Don't worry, that's completely normal!\n",
        "\n",
        "Just grab a snack and await the CHONK!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "collapsed": true,
        "id": "6qv6ArYrWKlu",
        "outputId": "daca2930-bdd0-46fc-f0c5-8722fbe4d199"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/1.2 MB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━━━━━━\u001b[0m \u001b[32m0.9/1.2 MB\u001b[0m \u001b[31m25.4 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m20.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ],
      "source": [
        "!pip install -q chonkie tiktoken docling model2vec vicinity together rich[jupyter]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "dODypx_0WKlv"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from typing import List\n",
        "\n",
        "from docling.document_converter import DocumentConverter\n",
        "from google.colab import userdata\n",
        "from model2vec import StaticModel\n",
        "from rich.console import Console\n",
        "from rich.text import Text\n",
        "from together import Together\n",
        "from transformers import AutoTokenizer\n",
        "from vicinity import Backend, Metric, Vicinity\n",
        "\n",
        "from chonkie import RecursiveChunker, RecursiveLevel, RecursiveRules"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kMTQUr2df8DD"
      },
      "source": [
        "## Inits and Utils\n",
        "\n",
        "Lets setup everything we need for a smooth chunking experience"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bv297-GIjHIr"
      },
      "source": [
        "### Define utilty functions\n",
        "\n",
        "Just some functions to make the `.pynb` experience better :D"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "PLEIT2gXgH8z"
      },
      "outputs": [],
      "source": [
        "# Rich text console for better printing\n",
        "console = Console()\n",
        "\n",
        "\n",
        "# A wrapper to pretty print\n",
        "def rprint(text: str, console: Console = console, width: int = 80) -> None:\n",
        "    richtext = Text(text)\n",
        "    console.print(richtext.wrap(console, width=width))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WYIMeyW8ipMH"
      },
      "source": [
        "### Initialize Your Model!\n",
        "\n",
        "Lets initialize the model we will build our application with!\n",
        "\n",
        "Here, we are going to be using Deepseek's R1 model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 336,
          "referenced_widgets": [
            "d0f8e826111040afa6edf4510aa49ca1",
            "81fa4c7d3afd420da2580d6355351e24",
            "9b9b2fc027ec4649bcd55a0e120b6cc7",
            "f9632096a6404732bda6026546266816",
            "16bfde6804e942ad954ddf88f5588454",
            "b6a13bbc38eb48539f65d9c644fac057",
            "41b95228bb5846708821b3fffa960483",
            "b044e5b76ff544bd84f807a6e84ce59f",
            "8cff535bdaa541bdb95bc83dfdeb17d5",
            "4fdbba77dced430aa48037d7854f5d9f",
            "35814869d9da480c8dbbdb93918e1ea2",
            "1b216830196c43b393a28db5bd765568",
            "3d1624fa6f434aeaaaffa8c0d1930849",
            "28ca59e41510442bac043477f7996f69",
            "25d57e49ed1f4df09338dd1b0b16f0c2",
            "a3fa8cd8020c49148b8bc02480253769",
            "47a364c06ce341ddb205a8dbf76d2c33",
            "225244cda4d84329bf3a625e94aa22d8",
            "d22f9517c502411ebe9c19860bf89fa4",
            "0bee3217e2cc4cf7becad3562bdf2244",
            "ff9a8e3024c548c3899da9b4dad4482b",
            "a9a2f6b4608b4c1cb080aa7b2af80f35",
            "7ab86a994d22489ebb6074f71ceb14bd",
            "a70961c198154a9a89924813c4e1efb7",
            "6edc3ad52fca48e7b1b4d89053598896",
            "580bb3ee8ae1462394b6f89c56081a3d",
            "20b14a12ed704bbc8679853441ef6c20",
            "926b01e2f8ea41d58171e20ad9cb9f22",
            "67668af85adf48449e8c786438b24cdd",
            "6a9dcd5dc6474161bb50fa2e449cff09",
            "9a49770411f94a34b7670dd39f25a2fa",
            "72aeb084f94a4bf4b9aa2b329c8670cc",
            "0014dbeaa5fa42a682780298dc3cd09e",
            "6be17e05e151422dbcb14ccc104accfa",
            "dace749f906b4eb98376a65be89372b0",
            "020957b384ba496aadde7977521bca30",
            "fbc977f7ba214ffab6de5580bcf27965",
            "60eced2d3ec045f8ae4a1ee38a4b0ca6",
            "ae764c73cbf94868aade3fe7887ee4fe",
            "a64575d99f3246519157b491213e9782",
            "3661db0317c24c778bbc469831294576",
            "8115608a9c7041c2b47ff07e17c103b9",
            "f765cdc5e04348c0af9eca910782a99e",
            "7fa65d8c9b30496881a4449aa2c4123a",
            "263b938af9ad47a1bff8ac95501d35b3",
            "a77ea7ec67ca4f3395bfae91142f1911",
            "c09535c24da547da976bb8b5be041882",
            "4d501fff616c49559ee4c2195682d1f2",
            "47d9410cc14241b8909a392a919b111a",
            "9d2d950283ca434cae4b121774e54497",
            "c205ad3ed80044eeb1025b2634e6c05b",
            "1958ea9a0d384a99bb812fda327df4e7",
            "13f4285fb4b949e281c41e9aacff1a98",
            "eb43d358ad1f406199cc34e9f8567660",
            "0e7234955eee4aaea6ecbb1dfd14dc74",
            "707c59c1b548468f91fcb09b2c6459aa",
            "1474cd1a4fc94614a31b94b98ebe7a7c",
            "f8c3ff6a8d024473abb5f7aa607cce64",
            "11eb446df0da421b92ad345124c1f873",
            "ed50674cbf90448196775c419b85356f",
            "2cd0482b91c04f34b374ae070b3a69ee",
            "51c53e3405ce4a779a022d7f0807e98e",
            "3e1500f646bc43a79d704ff2c40deed8",
            "43d38723da5f4b7d9a959b1b29cec782",
            "3aac3e5f2ea044cd906e1b3d5452b55d",
            "1dd4ab97ef784e628c9483582a4afa8b"
          ]
        },
        "collapsed": true,
        "id": "qCuTgkkkWKlw",
        "outputId": "e15e409a-e2f4-4d3d-acc3-db6058ea40b8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "model.safetensors:   0%|          | 0.00/129M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "d0f8e826111040afa6edf4510aa49ca1"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "README.md:   0%|          | 0.00/16.7k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "1b216830196c43b393a28db5bd765568"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "config.json:   0%|          | 0.00/202 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "7ab86a994d22489ebb6074f71ceb14bd"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer.json:   0%|          | 0.00/1.49M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "6be17e05e151422dbcb14ccc104accfa"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer_config.json:   0%|          | 0.00/3.59k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "263b938af9ad47a1bff8ac95501d35b3"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer.json:   0%|          | 0.00/7.85M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "707c59c1b548468f91fcb09b2c6459aa"
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Set your Together API key to use Deepseek R1 with it~\n",
        "os.environ[\"TOGETHER_API_KEY\"] = userdata.get(\"TOGETHER_API_KEY\")\n",
        "\n",
        "# Initialise a model2vec model for encoding sentences for retrieval\n",
        "model = StaticModel.from_pretrained(\"minishlab/potion-retrieval-32M\")\n",
        "\n",
        "# Initialise the Together client to call upon Deepseek R1\n",
        "client = Together()\n",
        "\n",
        "# (Optional) Initialise the tokenizer for Deepseek R1\n",
        "# We use this to get token counts at various points in this colab.\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"deepseek-ai/DeepSeek-R1\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B2DEzE8IWKlw"
      },
      "source": [
        "## Step 1: Use docling to convert from PDF to Markdown!\n",
        "\n",
        "**Note**: This step can take about 2-3 minutes depending on your PDF."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "collapsed": true,
        "id": "J5xeJIc8WKlw",
        "outputId": "f7679d2e-7f5f-43a1-a4c2-a568375d962a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:easyocr.easyocr:Downloading detection model, please wait. This may take several minutes depending upon your network connection.\n",
            "WARNING:easyocr.easyocr:Downloading recognition model, please wait. This may take several minutes depending upon your network connection.\n",
            "/usr/local/lib/python3.11/dist-packages/torch/utils/cpp_extension.py:1964: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. \n",
            "If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].\n",
            "  warnings.warn(\n"
          ]
        }
      ],
      "source": [
        "# Docling can convert any PDF to markdown!\n",
        "converter = DocumentConverter()\n",
        "source = \"https://arxiv.org/pdf/1706.03762\"\n",
        "result = converter.convert(source)\n",
        "text = result.document.export_to_markdown()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 859
        },
        "collapsed": true,
        "id": "qd9xcOVEWKlw",
        "outputId": "b9b589fa-6698-42e9-8518-015a88138da8"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Provided proper attribution is provided, Google hereby grants permission to \n",
              "reproduce the tables and figures in this paper solely for use in journalistic or\n",
              "scholarly works.\n",
              "\n",
              "## Attention Is All You Need\n",
              "\n",
              "Ashish Vaswani ∗ Google Brain avaswani@google.com\n",
              "\n",
              "Noam Shazeer ∗ Google Brain noam@google.com\n",
              "\n",
              "Niki Parmar ∗ Google Research nikip@google.com\n",
              "\n",
              "Jakob Uszkoreit ∗ Google Research usz@google.com\n",
              "\n",
              "Llion Jones ∗ Google Research llion@google.com\n",
              "\n",
              "Aidan N. Gomez ∗ † University of Toronto\n",
              "\n",
              "aidan@cs.toronto.edu\n",
              "\n",
              "Łukasz Kaiser Google Brain lukaszkaiser@google.com\n",
              "\n",
              "∗\n",
              "\n",
              "Illia Polosukhin ∗ ‡\n",
              "\n",
              "illia.polosukhin@gmail.com\n",
              "\n",
              "## Abstract\n",
              "\n",
              "The dominant sequence transduction models are based on complex recurrent or \n",
              "convolutional neural networks that include an encoder and a decoder. The best \n",
              "performing models also connect the encoder and decoder through an attention \n",
              "mechanism. We propose a new simple network architecture, the Transformer, based \n",
              "solely on attention mechanisms, dispensing with recurrence and convolutions \n",
              "entirely. Experiments on two machine translation tasks show these models to be \n",
              "superior in quality while being more parallelizable and requiring significantly \n",
              "less time to train. Our model achieves 28.4 BLEU on the WMT 2014 \n",
              "Englishto-German translation task, improving over the existing best results, \n",
              "including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French \n",
              "translation task, our model establishes a new single-model state-of-the-art BLEU\n",
              "score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the\n",
              "training costs of the best models from the literature. We show that the \n",
              "Transformer generalizes well to other tasks by applying it successfully to \n",
              "English constituency parsing both with large and limited training data.\n",
              "\n",
              "## 1 Introduction\n",
              "\n",
              "Recurrent neural networks, long short-term memory [13] and gated recurrent [7] \n",
              "neural networks in particular, have been firmly established as state of the art \n",
              "approaches in sequence modeling and transduction prob\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Provided proper attribution is provided, Google hereby grants permission to \n",
              "reproduce the tables and figures in this paper solely for use in journalistic or\n",
              "scholarly works.\n",
              "\n",
              "## Attention Is All You Need\n",
              "\n",
              "Ashish Vaswani ∗ Google Brain avaswani@google.com\n",
              "\n",
              "Noam Shazeer ∗ Google Brain noam@google.com\n",
              "\n",
              "Niki Parmar ∗ Google Research nikip@google.com\n",
              "\n",
              "Jakob Uszkoreit ∗ Google Research usz@google.com\n",
              "\n",
              "Llion Jones ∗ Google Research llion@google.com\n",
              "\n",
              "Aidan N. Gomez ∗ † University of Toronto\n",
              "\n",
              "aidan@cs.toronto.edu\n",
              "\n",
              "Łukasz Kaiser Google Brain lukaszkaiser@google.com\n",
              "\n",
              "∗\n",
              "\n",
              "Illia Polosukhin ∗ ‡\n",
              "\n",
              "illia.polosukhin@gmail.com\n",
              "\n",
              "## Abstract\n",
              "\n",
              "The dominant sequence transduction models are based on complex recurrent or \n",
              "convolutional neural networks that include an encoder and a decoder. The best \n",
              "performing models also connect the encoder and decoder through an attention \n",
              "mechanism. We propose a new simple network architecture, the Transformer, based \n",
              "solely on attention mechanisms, dispensing with recurrence and convolutions \n",
              "entirely. Experiments on two machine translation tasks show these models to be \n",
              "superior in quality while being more parallelizable and requiring significantly \n",
              "less time to train. Our model achieves 28.4 BLEU on the WMT 2014 \n",
              "Englishto-German translation task, improving over the existing best results, \n",
              "including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French \n",
              "translation task, our model establishes a new single-model state-of-the-art BLEU\n",
              "score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the\n",
              "training costs of the best models from the literature. We show that the \n",
              "Transformer generalizes well to other tasks by applying it successfully to \n",
              "English constituency parsing both with large and limited training data.\n",
              "\n",
              "## 1 Introduction\n",
              "\n",
              "Recurrent neural networks, long short-term memory [13] and gated recurrent [7] \n",
              "neural networks in particular, have been firmly established as state of the art \n",
              "approaches in sequence modeling and transduction prob\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "# @title A quick look at the text we'll be working with~\n",
        "rprint(text[:2000])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h-eUVqwAlHbv"
      },
      "source": [
        "### Get the total token counts for this PDF\n",
        "\n",
        "Our example PDF is made up of ~**9,865** tokens! Keep this in mind as we move forward ;)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "id": "pq4oUNuLWKlx",
        "outputId": "90b537c7-b787-44a1-b45d-8a18d9a8ed0e"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "This PDF contains: 9865 tokens\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">This PDF contains: 9865 tokens\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "total_text_tokens = len(tokenizer.encode(text))\n",
        "rprint(f\"This PDF contains: {total_text_tokens} tokens\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2HZUx_7XWKlx"
      },
      "source": [
        "## Step 2: Chunk your texts w/ Chonkie!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f999ufGNh8Fg"
      },
      "source": [
        "### Initalize your Chunker!\n",
        "For effective markdown chunking, we will be using Chonkie's **recurisve chunker**!\n",
        "\n",
        "With the recursive chunker, we can define custom RecursiveRules that fit our file's structure and format. Since we are working with Markdown files, we will specify rules that help Chonkie understand it's syntax."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "LgpFvIcmgfeo"
      },
      "outputs": [],
      "source": [
        "rules = RecursiveRules(\n",
        "    levels=[\n",
        "        RecursiveLevel(delimiters=[\"######\", \"#####\", \"####\", \"###\", \"##\", \"#\"], include_delim=\"next\"),\n",
        "        RecursiveLevel(delimiters=[\"\\n\\n\", \"\\n\", \"\\r\\n\", \"\\r\"]),\n",
        "        RecursiveLevel(delimiters=\".?!;:\"),\n",
        "        RecursiveLevel(),\n",
        "    ]\n",
        ")\n",
        "chunker = RecursiveChunker(rules=rules, chunk_size=384)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z8SIQmZhLlpI"
      },
      "source": [
        "### Lets Chunk!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fDhSfmHhWKly",
        "outputId": "4b445608-7ad6-4065-c0d1-9b3007473984"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total number of chunks: 57\n"
          ]
        }
      ],
      "source": [
        "# This is all it takes to chunk!\n",
        "chunks = chunker(text)\n",
        "print(f\"Total number of chunks: {len(chunks)}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "collapsed": true,
        "id": "SkUdtPdFWKly",
        "outputId": "74ad59fa-0956-4a93-ac7a-403c80dcb72d"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Provided proper attribution is provided, Google hereby grants permission to \n",
              "reproduce the tables and figures in this paper solely for use in journalistic or\n",
              "scholarly works.\n",
              "\n",
              "## Attention Is All You Need\n",
              "\n",
              "Ashish Vaswani ∗ Google Brain avaswani@google.com\n",
              "\n",
              "Noam Shazeer ∗ Google Brain noam@google.com\n",
              "\n",
              "Niki Parmar ∗ Google Research nikip@google.com\n",
              "\n",
              "Jakob Uszkoreit ∗ Google Research usz@google.com\n",
              "\n",
              "Llion Jones ∗ Google Research llion@google.com\n",
              "\n",
              "Aidan N. Gomez ∗ † University of Toronto\n",
              "\n",
              "aidan@cs.toronto.edu\n",
              "\n",
              "Łukasz Kaiser Google Brain lukaszkaiser@google.com\n",
              "\n",
              "∗\n",
              "\n",
              "Illia Polosukhin ∗ ‡\n",
              "\n",
              "illia.polosukhin@gmail.com\n",
              "\n",
              "\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Provided proper attribution is provided, Google hereby grants permission to \n",
              "reproduce the tables and figures in this paper solely for use in journalistic or\n",
              "scholarly works.\n",
              "\n",
              "## Attention Is All You Need\n",
              "\n",
              "Ashish Vaswani ∗ Google Brain avaswani@google.com\n",
              "\n",
              "Noam Shazeer ∗ Google Brain noam@google.com\n",
              "\n",
              "Niki Parmar ∗ Google Research nikip@google.com\n",
              "\n",
              "Jakob Uszkoreit ∗ Google Research usz@google.com\n",
              "\n",
              "Llion Jones ∗ Google Research llion@google.com\n",
              "\n",
              "Aidan N. Gomez ∗ † University of Toronto\n",
              "\n",
              "aidan@cs.toronto.edu\n",
              "\n",
              "Łukasz Kaiser Google Brain lukaszkaiser@google.com\n",
              "\n",
              "∗\n",
              "\n",
              "Illia Polosukhin ∗ ‡\n",
              "\n",
              "illia.polosukhin@gmail.com\n",
              "\n",
              "\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-------------------------------------------------------------------------------- \n",
            "\n",
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "## Abstract\n",
              "\n",
              "The dominant sequence transduction models are based on complex recurrent or \n",
              "convolutional neural networks that include an encoder and a decoder. The best \n",
              "performing models also connect the encoder and decoder through an attention \n",
              "mechanism. We propose a new simple network architecture, the Transformer, based \n",
              "solely on attention mechanisms, dispensing with recurrence and convolutions \n",
              "entirely. Experiments on two machine translation tasks show these models to be \n",
              "superior in quality while being more parallelizable and requiring significantly \n",
              "less time to train. Our model achieves 28.4 BLEU on the WMT 2014 \n",
              "Englishto-German translation task, improving over the existing best results, \n",
              "including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French \n",
              "translation task, our model establishes a new single-model state-of-the-art BLEU\n",
              "score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the\n",
              "training costs of the best models from the literature. We show that the \n",
              "Transformer generalizes well to other tasks by applying it successfully to \n",
              "English constituency parsing both with large and limited training data.\n",
              "\n",
              "\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">## Abstract\n",
              "\n",
              "The dominant sequence transduction models are based on complex recurrent or \n",
              "convolutional neural networks that include an encoder and a decoder. The best \n",
              "performing models also connect the encoder and decoder through an attention \n",
              "mechanism. We propose a new simple network architecture, the Transformer, based \n",
              "solely on attention mechanisms, dispensing with recurrence and convolutions \n",
              "entirely. Experiments on two machine translation tasks show these models to be \n",
              "superior in quality while being more parallelizable and requiring significantly \n",
              "less time to train. Our model achieves 28.4 BLEU on the WMT 2014 \n",
              "Englishto-German translation task, improving over the existing best results, \n",
              "including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French \n",
              "translation task, our model establishes a new single-model state-of-the-art BLEU\n",
              "score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the\n",
              "training costs of the best models from the literature. We show that the \n",
              "Transformer generalizes well to other tasks by applying it successfully to \n",
              "English constituency parsing both with large and limited training data.\n",
              "\n",
              "\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-------------------------------------------------------------------------------- \n",
            "\n",
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "## 1 Introduction\n",
              "\n",
              "Recurrent neural networks, long short-term memory [13] and gated recurrent [7] \n",
              "neural networks in particular, have been firmly established as state of the art \n",
              "approaches in sequence modeling and transduction problems such as language \n",
              "modeling and machine translation [35, 2, 5]. Numerous efforts have since \n",
              "continued to push the boundaries of recurrent language models and \n",
              "encoder-decoder architectures [38, 24, 15].\n",
              "\n",
              "Recurrent models typically factor computation along the symbol positions of the \n",
              "input and output sequences. Aligning the positions to steps in computation time,\n",
              "they generate a sequence of hidden states h t , as a function of the previous \n",
              "hidden state h t -1 and the input for position t . This inherently sequential \n",
              "nature precludes parallelization within training examples, which becomes \n",
              "critical at longer sequence lengths, as memory constraints limit batching across\n",
              "examples. Recent work has achieved significant improvements in computational \n",
              "efficiency through factorization tricks [21] and conditional computation [32], \n",
              "while also improving model performance in case of the latter. The fundamental \n",
              "constraint of sequential computation, however, remains.\n",
              "\n",
              "Attention mechanisms have become an integral part of compelling sequence \n",
              "modeling and transduction models in various tasks, allowing modeling of \n",
              "dependencies without regard to their distance in the input or output sequences \n",
              "[2, 19]. In all but a few cases [27], however, such attention mechanisms are \n",
              "used in conjunction with a recurrent network.\n",
              "\n",
              "In this work we propose the Transformer, a model architecture eschewing \n",
              "recurrence and instead relying entirely on an attention mechanism to draw global\n",
              "dependencies between input and output. The Transformer allows for significantly \n",
              "more parallelization and can reach a new state of the art in translation quality\n",
              "after being trained for as little as twelve hours on eight P100 GPUs.\n",
              "\n",
              "\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">## 1 Introduction\n",
              "\n",
              "Recurrent neural networks, long short-term memory [13] and gated recurrent [7] \n",
              "neural networks in particular, have been firmly established as state of the art \n",
              "approaches in sequence modeling and transduction problems such as language \n",
              "modeling and machine translation [35, 2, 5]. Numerous efforts have since \n",
              "continued to push the boundaries of recurrent language models and \n",
              "encoder-decoder architectures [38, 24, 15].\n",
              "\n",
              "Recurrent models typically factor computation along the symbol positions of the \n",
              "input and output sequences. Aligning the positions to steps in computation time,\n",
              "they generate a sequence of hidden states h t , as a function of the previous \n",
              "hidden state h t -1 and the input for position t . This inherently sequential \n",
              "nature precludes parallelization within training examples, which becomes \n",
              "critical at longer sequence lengths, as memory constraints limit batching across\n",
              "examples. Recent work has achieved significant improvements in computational \n",
              "efficiency through factorization tricks [21] and conditional computation [32], \n",
              "while also improving model performance in case of the latter. The fundamental \n",
              "constraint of sequential computation, however, remains.\n",
              "\n",
              "Attention mechanisms have become an integral part of compelling sequence \n",
              "modeling and transduction models in various tasks, allowing modeling of \n",
              "dependencies without regard to their distance in the input or output sequences \n",
              "[2, 19]. In all but a few cases [27], however, such attention mechanisms are \n",
              "used in conjunction with a recurrent network.\n",
              "\n",
              "In this work we propose the Transformer, a model architecture eschewing \n",
              "recurrence and instead relying entirely on an attention mechanism to draw global\n",
              "dependencies between input and output. The Transformer allows for significantly \n",
              "more parallelization and can reach a new state of the art in translation quality\n",
              "after being trained for as little as twelve hours on eight P100 GPUs.\n",
              "\n",
              "\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-------------------------------------------------------------------------------- \n",
            "\n",
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "## 2 Background\n",
              "\n",
              "The goal of reducing sequential computation also forms the foundation of the \n",
              "Extended Neural GPU [16], ByteNet [18] and ConvS2S [9], all of which use \n",
              "convolutional neural networks as basic building block, computing hidden \n",
              "representations in parallel for all input and output positions. In these models,\n",
              "the number of operations required to relate signals from two arbitrary input or \n",
              "output positions grows in the distance between positions, linearly for ConvS2S \n",
              "and logarithmically for ByteNet. This makes it more difficult to learn \n",
              "dependencies between distant positions [12]. In the Transformer this is reduced \n",
              "to a constant number of operations, albeit at the cost of reduced effective \n",
              "resolution due to averaging attention-weighted positions, an effect we \n",
              "counteract with Multi-Head Attention as described in section 3.2.\n",
              "\n",
              "Self-attention, sometimes called intra-attention is an attention mechanism \n",
              "relating different positions of a single sequence in order to compute a \n",
              "representation of the sequence. Self-attention has been used successfully in a \n",
              "variety of tasks including reading comprehension, abstractive summarization, \n",
              "textual entailment and learning task-independent sentence representations [4, \n",
              "27, 28, 22].\n",
              "\n",
              "End-to-end memory networks are based on a recurrent attention mechanism instead \n",
              "of sequencealigned recurrence and have been shown to perform well on \n",
              "simple-language question answering and language modeling tasks [34].\n",
              "\n",
              "To the best of our knowledge, however, the Transformer is the first transduction\n",
              "model relying entirely on self-attention to compute representations of its input\n",
              "and output without using sequencealigned RNNs or convolution. In the following \n",
              "sections, we will describe the Transformer, motivate self-attention and discuss \n",
              "its advantages over models such as [17, 18] and [9].\n",
              "\n",
              "\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">## 2 Background\n",
              "\n",
              "The goal of reducing sequential computation also forms the foundation of the \n",
              "Extended Neural GPU [16], ByteNet [18] and ConvS2S [9], all of which use \n",
              "convolutional neural networks as basic building block, computing hidden \n",
              "representations in parallel for all input and output positions. In these models,\n",
              "the number of operations required to relate signals from two arbitrary input or \n",
              "output positions grows in the distance between positions, linearly for ConvS2S \n",
              "and logarithmically for ByteNet. This makes it more difficult to learn \n",
              "dependencies between distant positions [12]. In the Transformer this is reduced \n",
              "to a constant number of operations, albeit at the cost of reduced effective \n",
              "resolution due to averaging attention-weighted positions, an effect we \n",
              "counteract with Multi-Head Attention as described in section 3.2.\n",
              "\n",
              "Self-attention, sometimes called intra-attention is an attention mechanism \n",
              "relating different positions of a single sequence in order to compute a \n",
              "representation of the sequence. Self-attention has been used successfully in a \n",
              "variety of tasks including reading comprehension, abstractive summarization, \n",
              "textual entailment and learning task-independent sentence representations [4, \n",
              "27, 28, 22].\n",
              "\n",
              "End-to-end memory networks are based on a recurrent attention mechanism instead \n",
              "of sequencealigned recurrence and have been shown to perform well on \n",
              "simple-language question answering and language modeling tasks [34].\n",
              "\n",
              "To the best of our knowledge, however, the Transformer is the first transduction\n",
              "model relying entirely on self-attention to compute representations of its input\n",
              "and output without using sequencealigned RNNs or convolution. In the following \n",
              "sections, we will describe the Transformer, motivate self-attention and discuss \n",
              "its advantages over models such as [17, 18] and [9].\n",
              "\n",
              "\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-------------------------------------------------------------------------------- \n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# @title A quick look at our chunks~\n",
        "for chunk in chunks[:4]:\n",
        "    rprint(chunk.text)\n",
        "    print(\"-\" * 80, \"\\n\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WmY3lMVxWKly"
      },
      "source": [
        "## Step 3: Retrieval Augmented Generation!\n",
        "\n",
        "Let's setup RAG to get our chunks when needed"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UzyJMtLUWKly"
      },
      "source": [
        "### Step 3.1: Get the embeddings for each of the chunks!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-VvbKceEWKly",
        "outputId": "8e717b86-b23d-4ef2-8c06-3cc1ed5e70f4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(57, 512)\n"
          ]
        }
      ],
      "source": [
        "items = [chunk.text for chunk in chunks]\n",
        "vectors = model.encode(items)\n",
        "print(vectors.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1EaYeq7IWKlz"
      },
      "source": [
        "### Step 3.2: Create an index with the chunks and embeddings for retrieval"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "KdaHRMi3WKlz"
      },
      "outputs": [],
      "source": [
        "# Initialize the Vicinity instance (using basic backend and cosine metric)\n",
        "vicinity = Vicinity.from_vectors_and_items(\n",
        "    vectors=vectors, items=items, backend_type=Backend.BASIC, metric=Metric.COSINE\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6SJjXWZNWKlz"
      },
      "source": [
        "### Step 3.3: Pack it all together in a single retrieval function!\n",
        "\n",
        "Given a query, retrieve all relevant embeddings~"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "2AYHiaWPteUO"
      },
      "outputs": [],
      "source": [
        "def get_embeddings(query: str):\n",
        "    query_vector = model.encode(query)\n",
        "    results = vicinity.query(query_vector, k=4)\n",
        "    return [x[0] for x in results[0]]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_slMghH3tt_3"
      },
      "source": [
        "### (Optional) Step 3.4: Test our function"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "PLENTTzMWKlz",
        "outputId": "43b6057f-7ad1-4652-9550-63384a63dd61"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "## Attention Visualizations Input-Input Layer5\n",
              "\n",
              "Figure 3: An example of the attention mechanism following long-distance \n",
              "dependencies in the encoder self-attention in layer 5 of 6. Many of the \n",
              "attention heads attend to a distant dependency of the verb 'making', completing \n",
              "the phrase 'making...more difficult'. Attentions here shown only for the word \n",
              "'making'. Different colors represent different heads. Best viewed in color.\n",
              "\n",
              "<!-- image -->\n",
              "\n",
              "Input-Input Layer5\n",
              "\n",
              "Figure 4: Two attention heads, also in layer 5 of 6, apparently involved in \n",
              "anaphora resolution. Top: Full attentions for head 5. Bottom: Isolated \n",
              "attentions from just the word 'its' for attention heads 5 and 6. Note that the \n",
              "attentions are very sharp for this word.\n",
              "\n",
              "<!-- image -->\n",
              "\n",
              "Input-Input Layer5\n",
              "\n",
              "Figure 5: Many of the attention heads exhibit behaviour that seems related to \n",
              "the structure of the sentence. We give two such examples above, from two \n",
              "different heads from the encoder self-attention at layer 5 of 6. The heads \n",
              "clearly learned to perform different tasks.\n",
              "\n",
              "<!-- image -->\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">## Attention Visualizations Input-Input Layer5\n",
              "\n",
              "Figure 3: An example of the attention mechanism following long-distance \n",
              "dependencies in the encoder self-attention in layer 5 of 6. Many of the \n",
              "attention heads attend to a distant dependency of the verb 'making', completing \n",
              "the phrase 'making...more difficult'. Attentions here shown only for the word \n",
              "'making'. Different colors represent different heads. Best viewed in color.\n",
              "\n",
              "&lt;!-- image --&gt;\n",
              "\n",
              "Input-Input Layer5\n",
              "\n",
              "Figure 4: Two attention heads, also in layer 5 of 6, apparently involved in \n",
              "anaphora resolution. Top: Full attentions for head 5. Bottom: Isolated \n",
              "attentions from just the word 'its' for attention heads 5 and 6. Note that the \n",
              "attentions are very sharp for this word.\n",
              "\n",
              "&lt;!-- image --&gt;\n",
              "\n",
              "Input-Input Layer5\n",
              "\n",
              "Figure 5: Many of the attention heads exhibit behaviour that seems related to \n",
              "the structure of the sentence. We give two such examples above, from two \n",
              "different heads from the encoder self-attention at layer 5 of 6. The heads \n",
              "clearly learned to perform different tasks.\n",
              "\n",
              "&lt;!-- image --&gt;\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-------------------------------------------------------------------------------- \n",
            "\n",
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "## 2 Background\n",
              "\n",
              "The goal of reducing sequential computation also forms the foundation of the \n",
              "Extended Neural GPU [16], ByteNet [18] and ConvS2S [9], all of which use \n",
              "convolutional neural networks as basic building block, computing hidden \n",
              "representations in parallel for all input and output positions. In these models,\n",
              "the number of operations required to relate signals from two arbitrary input or \n",
              "output positions grows in the distance between positions, linearly for ConvS2S \n",
              "and logarithmically for ByteNet. This makes it more difficult to learn \n",
              "dependencies between distant positions [12]. In the Transformer this is reduced \n",
              "to a constant number of operations, albeit at the cost of reduced effective \n",
              "resolution due to averaging attention-weighted positions, an effect we \n",
              "counteract with Multi-Head Attention as described in section 3.2.\n",
              "\n",
              "Self-attention, sometimes called intra-attention is an attention mechanism \n",
              "relating different positions of a single sequence in order to compute a \n",
              "representation of the sequence. Self-attention has been used successfully in a \n",
              "variety of tasks including reading comprehension, abstractive summarization, \n",
              "textual entailment and learning task-independent sentence representations [4, \n",
              "27, 28, 22].\n",
              "\n",
              "End-to-end memory networks are based on a recurrent attention mechanism instead \n",
              "of sequencealigned recurrence and have been shown to perform well on \n",
              "simple-language question answering and language modeling tasks [34].\n",
              "\n",
              "To the best of our knowledge, however, the Transformer is the first transduction\n",
              "model relying entirely on self-attention to compute representations of its input\n",
              "and output without using sequencealigned RNNs or convolution. In the following \n",
              "sections, we will describe the Transformer, motivate self-attention and discuss \n",
              "its advantages over models such as [17, 18] and [9].\n",
              "\n",
              "\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">## 2 Background\n",
              "\n",
              "The goal of reducing sequential computation also forms the foundation of the \n",
              "Extended Neural GPU [16], ByteNet [18] and ConvS2S [9], all of which use \n",
              "convolutional neural networks as basic building block, computing hidden \n",
              "representations in parallel for all input and output positions. In these models,\n",
              "the number of operations required to relate signals from two arbitrary input or \n",
              "output positions grows in the distance between positions, linearly for ConvS2S \n",
              "and logarithmically for ByteNet. This makes it more difficult to learn \n",
              "dependencies between distant positions [12]. In the Transformer this is reduced \n",
              "to a constant number of operations, albeit at the cost of reduced effective \n",
              "resolution due to averaging attention-weighted positions, an effect we \n",
              "counteract with Multi-Head Attention as described in section 3.2.\n",
              "\n",
              "Self-attention, sometimes called intra-attention is an attention mechanism \n",
              "relating different positions of a single sequence in order to compute a \n",
              "representation of the sequence. Self-attention has been used successfully in a \n",
              "variety of tasks including reading comprehension, abstractive summarization, \n",
              "textual entailment and learning task-independent sentence representations [4, \n",
              "27, 28, 22].\n",
              "\n",
              "End-to-end memory networks are based on a recurrent attention mechanism instead \n",
              "of sequencealigned recurrence and have been shown to perform well on \n",
              "simple-language question answering and language modeling tasks [34].\n",
              "\n",
              "To the best of our knowledge, however, the Transformer is the first transduction\n",
              "model relying entirely on self-attention to compute representations of its input\n",
              "and output without using sequencealigned RNNs or convolution. In the following \n",
              "sections, we will describe the Transformer, motivate self-attention and discuss \n",
              "its advantages over models such as [17, 18] and [9].\n",
              "\n",
              "\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-------------------------------------------------------------------------------- \n",
            "\n",
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "## Scaled Dot-Product Attention\n",
              "\n",
              "<!-- image -->\n",
              "\n",
              "Figure 2: (left) Scaled Dot-Product Attention. (right) Multi-Head Attention \n",
              "consists of several attention layers running in parallel.\n",
              "\n",
              "<!-- image -->\n",
              "\n",
              "of the values, where the weight assigned to each value is computed by a \n",
              "compatibility function of the query with the corresponding key.\n",
              "\n",
              "\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">## Scaled Dot-Product Attention\n",
              "\n",
              "&lt;!-- image --&gt;\n",
              "\n",
              "Figure 2: (left) Scaled Dot-Product Attention. (right) Multi-Head Attention \n",
              "consists of several attention layers running in parallel.\n",
              "\n",
              "&lt;!-- image --&gt;\n",
              "\n",
              "of the values, where the weight assigned to each value is computed by a \n",
              "compatibility function of the query with the corresponding key.\n",
              "\n",
              "\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-------------------------------------------------------------------------------- \n",
            "\n",
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "## 3.2.2 Multi-Head Attention\n",
              "\n",
              "Instead of performing a single attention function with d model-dimensional keys,\n",
              "values and queries, we found it beneficial to linearly project the queries, keys\n",
              "and values h times with different, learned linear projections to d k , d k and d\n",
              "v dimensions, respectively. On each of these projected versions of queries, keys\n",
              "and values we then perform the attention function in parallel, yielding d v \n",
              "-dimensional\n",
              "\n",
              "output values. These are concatenated and once again projected, resulting in the\n",
              "final values, as depicted in Figure 2.\n",
              "\n",
              "Multi-head attention allows the model to jointly attend to information from \n",
              "different representation subspaces at different positions. With a single \n",
              "attention head, averaging inhibits this.\n",
              "\n",
              "<!-- formula-not-decoded -->\n",
              "\n",
              "<!-- formula-not-decoded -->\n",
              "\n",
              "Where the projections are parameter matrices W Q i ∈ R d model × d k , W K i ∈ R\n",
              "d model × d k , W V i ∈ R d model × d v and W O ∈ R hd v × d model .\n",
              "\n",
              "In this work we employ h = 8 parallel attention layers, or heads. For each of \n",
              "these we use d k = d v = d model /h = 64 . Due to the reduced dimension of each \n",
              "head, the total computational cost is similar to that of single-head attention \n",
              "with full dimensionality.\n",
              "\n",
              "\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">## 3.2.2 Multi-Head Attention\n",
              "\n",
              "Instead of performing a single attention function with d model-dimensional keys,\n",
              "values and queries, we found it beneficial to linearly project the queries, keys\n",
              "and values h times with different, learned linear projections to d k , d k and d\n",
              "v dimensions, respectively. On each of these projected versions of queries, keys\n",
              "and values we then perform the attention function in parallel, yielding d v \n",
              "-dimensional\n",
              "\n",
              "output values. These are concatenated and once again projected, resulting in the\n",
              "final values, as depicted in Figure 2.\n",
              "\n",
              "Multi-head attention allows the model to jointly attend to information from \n",
              "different representation subspaces at different positions. With a single \n",
              "attention head, averaging inhibits this.\n",
              "\n",
              "&lt;!-- formula-not-decoded --&gt;\n",
              "\n",
              "&lt;!-- formula-not-decoded --&gt;\n",
              "\n",
              "Where the projections are parameter matrices W Q i ∈ R d model × d k , W K i ∈ R\n",
              "d model × d k , W V i ∈ R d model × d v and W O ∈ R hd v × d model .\n",
              "\n",
              "In this work we employ h = 8 parallel attention layers, or heads. For each of \n",
              "these we use d k = d v = d model /h = 64 . Due to the reduced dimension of each \n",
              "head, the total computational cost is similar to that of single-head attention \n",
              "with full dimensionality.\n",
              "\n",
              "\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-------------------------------------------------------------------------------- \n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "query = \"What is a Multi-Head Self Attention?\"\n",
        "retrieved_chunks = get_embeddings(query)\n",
        "\n",
        "for chunk in retrieved_chunks:\n",
        "    rprint(chunk)\n",
        "    print(\"-\" * 80, \"\\n\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YPy7bqrEWKl0"
      },
      "source": [
        "## Step 4: Lets build our LLM Application!\n",
        "\n",
        "Our data is ready! Let's setup our LLM application to answer user queries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "CRjaTG2oR7eC"
      },
      "outputs": [],
      "source": [
        "# A simple function to make LLM prompts with chunks\n",
        "def create_prompt(chunks: List[str], query: str) -> str:\n",
        "    prompt_template = \"\"\"<instructions>\n",
        "  Based on the provided contexts, answer the given question to the best of your ability. Remember to also add citations at appropriate points in the format of square brackets like [1][2][3], especially at sentence or paragraph endings.\n",
        "  You will be given 4 passages in the context, marked with a label 'Doc [1]:' to denote the passage number. Use that number for citations. Answer only from the given context, and if there's no appropriate context, reply \"No relevant context found!\".\n",
        "  </instructions>\n",
        "\n",
        "  <context>\n",
        "  {context}\n",
        "  </context>\n",
        "\n",
        "  <query>\n",
        "  {query}\n",
        "  </query>\n",
        "  \"\"\"\n",
        "    context = \"\\n\\n\".join([f\"Doc {i+1}: {chunk}\" for i, chunk in enumerate(chunks)])\n",
        "    prompt = prompt_template.format(context=context, query=query)\n",
        "    return prompt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "k40GjsL2WKl0"
      },
      "outputs": [],
      "source": [
        "# Prompt to use\n",
        "query = \"What is a Multi-Head Self Attention?\"\n",
        "retrieved_chunks = get_embeddings(query)\n",
        "prompt = create_prompt(retrieved_chunks, query)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 298
        },
        "id": "BnY1fZzEWKl0",
        "outputId": "b69bff38-a1fc-45ae-bba0-30174972e6db"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\n",
              "\n",
              "Multi-Head Self Attention is a mechanism in the Transformer model that enables \n",
              "the system to jointly attend to information from different representation \n",
              "subspaces at different positions. Instead of performing a single attention \n",
              "function, it linearly projects the queries, keys, and values multiple times \n",
              "(using distinct learned projections for each \"head\") and executes the scaled \n",
              "dot-product attention function in parallel across these projections [4]. The \n",
              "outputs of these parallel attention layers are concatenated and reprojected to \n",
              "form the final result [4]. \n",
              "\n",
              "This approach allows the model to capture diverse dependencies, such as \n",
              "long-distance syntactic relationships (e.g., resolving anaphora like \"its\" in \n",
              "Figure 4) or structural patterns in sentences (Figure 5), as demonstrated in \n",
              "encoder self-attention layers [1]. By using multiple heads (e.g., 8 heads with \n",
              "reduced dimensionality per head), the Transformer maintains computational \n",
              "efficiency while mitigating the reduced resolution caused by averaging in \n",
              "single-head attention [4][2].\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
              "\n",
              "Multi-Head Self Attention is a mechanism in the Transformer model that enables \n",
              "the system to jointly attend to information from different representation \n",
              "subspaces at different positions. Instead of performing a single attention \n",
              "function, it linearly projects the queries, keys, and values multiple times \n",
              "(using distinct learned projections for each \"head\") and executes the scaled \n",
              "dot-product attention function in parallel across these projections [4]. The \n",
              "outputs of these parallel attention layers are concatenated and reprojected to \n",
              "form the final result [4]. \n",
              "\n",
              "This approach allows the model to capture diverse dependencies, such as \n",
              "long-distance syntactic relationships (e.g., resolving anaphora like \"its\" in \n",
              "Figure 4) or structural patterns in sentences (Figure 5), as demonstrated in \n",
              "encoder self-attention layers [1]. By using multiple heads (e.g., 8 heads with \n",
              "reduced dimensionality per head), the Transformer maintains computational \n",
              "efficiency while mitigating the reduced resolution caused by averaging in \n",
              "single-head attention [4][2].\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "response = client.chat.completions.create(\n",
        "    model=\"deepseek-ai/DeepSeek-R1\",\n",
        "    messages=[{\"role\": \"user\", \"content\": prompt}],\n",
        ")\n",
        "# Print the final response without the thinking tokens\n",
        "answer = response.choices[0].message.content.split(\"</think>\")[-1]\n",
        "rprint(answer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jS9byevRO3KQ"
      },
      "source": [
        "### Wow, the model works great! And how many tokens did we use?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "id": "JciPBJ29WKl1",
        "outputId": "b95afe9b-e87a-413c-b4e7-5bc61d1f76da"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "This prompt contains: 1067 tokens\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">This prompt contains: 1067 tokens\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "prompt_tokens = len(tokenizer.encode(prompt))\n",
        "rprint(f\"This prompt contains: {prompt_tokens} tokens\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K9A5B81FPFnq"
      },
      "source": [
        "**1,069 tokens!** Rememeber from earlier--without chunking we'd have used 9,865 tokens!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cyaRx7JPWKl1"
      },
      "source": [
        "## Conclusions\n",
        "\n",
        "We got a great answer back from our model, while saving about **~8,800 input tokens**! Chonkie is one efficient hippo 🚀\n",
        "\n",
        "Hope you found this notebook useful! If you want to learn more about Chonkie, check our [GitHub](https://github.com/chonkie-ai/chonkie), [Twitter](https://x.com/ChonkieAI), and [Bluesky](https://bsky.app/profile/chonkieai.bsky.social)!\n",
        "\n",
        "If you have any questions about anything we showed us here, reach us at `support@chonkie.ai` or on our [Discord](https://discord.gg/6V5pqvqsCY)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wy7Sa4KMaUss"
      },
      "source": [
        "# Fin\n",
        "\n",
        "Happy Chonking! 🦛✨"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "d0f8e826111040afa6edf4510aa49ca1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_81fa4c7d3afd420da2580d6355351e24",
              "IPY_MODEL_9b9b2fc027ec4649bcd55a0e120b6cc7",
              "IPY_MODEL_f9632096a6404732bda6026546266816"
            ],
            "layout": "IPY_MODEL_16bfde6804e942ad954ddf88f5588454"
          }
        },
        "81fa4c7d3afd420da2580d6355351e24": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b6a13bbc38eb48539f65d9c644fac057",
            "placeholder": "​",
            "style": "IPY_MODEL_41b95228bb5846708821b3fffa960483",
            "value": "model.safetensors: 100%"
          }
        },
        "9b9b2fc027ec4649bcd55a0e120b6cc7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b044e5b76ff544bd84f807a6e84ce59f",
            "max": 129210456,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_8cff535bdaa541bdb95bc83dfdeb17d5",
            "value": 129210456
          }
        },
        "f9632096a6404732bda6026546266816": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4fdbba77dced430aa48037d7854f5d9f",
            "placeholder": "​",
            "style": "IPY_MODEL_35814869d9da480c8dbbdb93918e1ea2",
            "value": " 129M/129M [00:03&lt;00:00, 42.3MB/s]"
          }
        },
        "16bfde6804e942ad954ddf88f5588454": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b6a13bbc38eb48539f65d9c644fac057": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "41b95228bb5846708821b3fffa960483": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b044e5b76ff544bd84f807a6e84ce59f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8cff535bdaa541bdb95bc83dfdeb17d5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "4fdbba77dced430aa48037d7854f5d9f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "35814869d9da480c8dbbdb93918e1ea2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1b216830196c43b393a28db5bd765568": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3d1624fa6f434aeaaaffa8c0d1930849",
              "IPY_MODEL_28ca59e41510442bac043477f7996f69",
              "IPY_MODEL_25d57e49ed1f4df09338dd1b0b16f0c2"
            ],
            "layout": "IPY_MODEL_a3fa8cd8020c49148b8bc02480253769"
          }
        },
        "3d1624fa6f434aeaaaffa8c0d1930849": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_47a364c06ce341ddb205a8dbf76d2c33",
            "placeholder": "​",
            "style": "IPY_MODEL_225244cda4d84329bf3a625e94aa22d8",
            "value": "README.md: 100%"
          }
        },
        "28ca59e41510442bac043477f7996f69": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d22f9517c502411ebe9c19860bf89fa4",
            "max": 16673,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0bee3217e2cc4cf7becad3562bdf2244",
            "value": 16673
          }
        },
        "25d57e49ed1f4df09338dd1b0b16f0c2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ff9a8e3024c548c3899da9b4dad4482b",
            "placeholder": "​",
            "style": "IPY_MODEL_a9a2f6b4608b4c1cb080aa7b2af80f35",
            "value": " 16.7k/16.7k [00:00&lt;00:00, 1.32MB/s]"
          }
        },
        "a3fa8cd8020c49148b8bc02480253769": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "47a364c06ce341ddb205a8dbf76d2c33": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "225244cda4d84329bf3a625e94aa22d8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d22f9517c502411ebe9c19860bf89fa4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0bee3217e2cc4cf7becad3562bdf2244": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "ff9a8e3024c548c3899da9b4dad4482b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a9a2f6b4608b4c1cb080aa7b2af80f35": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7ab86a994d22489ebb6074f71ceb14bd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_a70961c198154a9a89924813c4e1efb7",
              "IPY_MODEL_6edc3ad52fca48e7b1b4d89053598896",
              "IPY_MODEL_580bb3ee8ae1462394b6f89c56081a3d"
            ],
            "layout": "IPY_MODEL_20b14a12ed704bbc8679853441ef6c20"
          }
        },
        "a70961c198154a9a89924813c4e1efb7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_926b01e2f8ea41d58171e20ad9cb9f22",
            "placeholder": "​",
            "style": "IPY_MODEL_67668af85adf48449e8c786438b24cdd",
            "value": "config.json: 100%"
          }
        },
        "6edc3ad52fca48e7b1b4d89053598896": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6a9dcd5dc6474161bb50fa2e449cff09",
            "max": 202,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_9a49770411f94a34b7670dd39f25a2fa",
            "value": 202
          }
        },
        "580bb3ee8ae1462394b6f89c56081a3d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_72aeb084f94a4bf4b9aa2b329c8670cc",
            "placeholder": "​",
            "style": "IPY_MODEL_0014dbeaa5fa42a682780298dc3cd09e",
            "value": " 202/202 [00:00&lt;00:00, 15.4kB/s]"
          }
        },
        "20b14a12ed704bbc8679853441ef6c20": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "926b01e2f8ea41d58171e20ad9cb9f22": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "67668af85adf48449e8c786438b24cdd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6a9dcd5dc6474161bb50fa2e449cff09": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9a49770411f94a34b7670dd39f25a2fa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "72aeb084f94a4bf4b9aa2b329c8670cc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0014dbeaa5fa42a682780298dc3cd09e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6be17e05e151422dbcb14ccc104accfa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_dace749f906b4eb98376a65be89372b0",
              "IPY_MODEL_020957b384ba496aadde7977521bca30",
              "IPY_MODEL_fbc977f7ba214ffab6de5580bcf27965"
            ],
            "layout": "IPY_MODEL_60eced2d3ec045f8ae4a1ee38a4b0ca6"
          }
        },
        "dace749f906b4eb98376a65be89372b0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ae764c73cbf94868aade3fe7887ee4fe",
            "placeholder": "​",
            "style": "IPY_MODEL_a64575d99f3246519157b491213e9782",
            "value": "tokenizer.json: 100%"
          }
        },
        "020957b384ba496aadde7977521bca30": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3661db0317c24c778bbc469831294576",
            "max": 1493150,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_8115608a9c7041c2b47ff07e17c103b9",
            "value": 1493150
          }
        },
        "fbc977f7ba214ffab6de5580bcf27965": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f765cdc5e04348c0af9eca910782a99e",
            "placeholder": "​",
            "style": "IPY_MODEL_7fa65d8c9b30496881a4449aa2c4123a",
            "value": " 1.49M/1.49M [00:00&lt;00:00, 8.07MB/s]"
          }
        },
        "60eced2d3ec045f8ae4a1ee38a4b0ca6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ae764c73cbf94868aade3fe7887ee4fe": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a64575d99f3246519157b491213e9782": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "3661db0317c24c778bbc469831294576": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8115608a9c7041c2b47ff07e17c103b9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "f765cdc5e04348c0af9eca910782a99e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7fa65d8c9b30496881a4449aa2c4123a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "263b938af9ad47a1bff8ac95501d35b3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_a77ea7ec67ca4f3395bfae91142f1911",
              "IPY_MODEL_c09535c24da547da976bb8b5be041882",
              "IPY_MODEL_4d501fff616c49559ee4c2195682d1f2"
            ],
            "layout": "IPY_MODEL_47d9410cc14241b8909a392a919b111a"
          }
        },
        "a77ea7ec67ca4f3395bfae91142f1911": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9d2d950283ca434cae4b121774e54497",
            "placeholder": "​",
            "style": "IPY_MODEL_c205ad3ed80044eeb1025b2634e6c05b",
            "value": "tokenizer_config.json: 100%"
          }
        },
        "c09535c24da547da976bb8b5be041882": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1958ea9a0d384a99bb812fda327df4e7",
            "max": 3594,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_13f4285fb4b949e281c41e9aacff1a98",
            "value": 3594
          }
        },
        "4d501fff616c49559ee4c2195682d1f2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_eb43d358ad1f406199cc34e9f8567660",
            "placeholder": "​",
            "style": "IPY_MODEL_0e7234955eee4aaea6ecbb1dfd14dc74",
            "value": " 3.59k/3.59k [00:00&lt;00:00, 212kB/s]"
          }
        },
        "47d9410cc14241b8909a392a919b111a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9d2d950283ca434cae4b121774e54497": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c205ad3ed80044eeb1025b2634e6c05b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1958ea9a0d384a99bb812fda327df4e7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "13f4285fb4b949e281c41e9aacff1a98": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "eb43d358ad1f406199cc34e9f8567660": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0e7234955eee4aaea6ecbb1dfd14dc74": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "707c59c1b548468f91fcb09b2c6459aa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_1474cd1a4fc94614a31b94b98ebe7a7c",
              "IPY_MODEL_f8c3ff6a8d024473abb5f7aa607cce64",
              "IPY_MODEL_11eb446df0da421b92ad345124c1f873"
            ],
            "layout": "IPY_MODEL_ed50674cbf90448196775c419b85356f"
          }
        },
        "1474cd1a4fc94614a31b94b98ebe7a7c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2cd0482b91c04f34b374ae070b3a69ee",
            "placeholder": "​",
            "style": "IPY_MODEL_51c53e3405ce4a779a022d7f0807e98e",
            "value": "tokenizer.json: 100%"
          }
        },
        "f8c3ff6a8d024473abb5f7aa607cce64": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3e1500f646bc43a79d704ff2c40deed8",
            "max": 7847602,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_43d38723da5f4b7d9a959b1b29cec782",
            "value": 7847602
          }
        },
        "11eb446df0da421b92ad345124c1f873": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3aac3e5f2ea044cd906e1b3d5452b55d",
            "placeholder": "​",
            "style": "IPY_MODEL_1dd4ab97ef784e628c9483582a4afa8b",
            "value": " 7.85M/7.85M [00:00&lt;00:00, 25.2MB/s]"
          }
        },
        "ed50674cbf90448196775c419b85356f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2cd0482b91c04f34b374ae070b3a69ee": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "51c53e3405ce4a779a022d7f0807e98e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "3e1500f646bc43a79d704ff2c40deed8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "43d38723da5f4b7d9a959b1b29cec782": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "3aac3e5f2ea044cd906e1b3d5452b55d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1dd4ab97ef784e628c9483582a4afa8b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}